[BugFix] Seperate prometheus multiproc dir for single-server multi-dp services#8059
[BugFix] Seperate prometheus multiproc dir for single-server multi-dp services#8059liyonghua0910 wants to merge 4 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8059 +/- ##
==========================================
Coverage ? 67.49%
==========================================
Files ? 475
Lines ? 66955
Branches ? 10332
==========================================
Hits ? 45193
Misses ? 18888
Partials ? 2874
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
liyonghua0910
force-pushed
the
develop+20260616_fix_dp_metrics
branch
from
d1ec8f0 to
15b4d28
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
CI报告基于以下代码生成（30分钟更新一次）: 1 Required任务 : 9/10 通过
2 失败详情🔴 Approval — 需要 Approval（置信度: 高）该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。 修复建议：请通过人工审批。 |
liyonghua0910
force-pushed
the
develop+20260616_fix_dp_metrics
branch
from
15b4d28 to
13e5962
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
liyonghua0910
force-pushed
the
develop+20260616_fix_dp_metrics
branch
from
13e5962 to
9a08ce4
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-07-03 19:33:30
📋 Review 摘要
PR 概述：为 Prometheus multiprocess 目录增加按 DP rank 隔离逻辑，覆盖 engine/internal-adapter 和 multi_api_server 场景
变更范围：fastdeploy/metrics/、fastdeploy/engine/、fastdeploy/entrypoints/openai/ 及相关单测
影响面 Tag：[Engine] [APIServer]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🔴 Bug | fastdeploy/entrypoints/openai/multi_api_server.py:112 |
multi_api_server 复用会迁移当前进程 .db 的 helper，可能把 supervisor 指标混入 DP0 |
📝 PR 规范检查
符合规范。
总体评价
按 DP 拆目录方向正确，但 multi_api_server 是 subprocess 启动模型，与 engine 的 fork/当前进程切换语义不同。需要避免在只准备子进程 env 时搬迁父进程 Prometheus 文件，否则 DP0 metrics 仍会被污染。
| os.makedirs(prom_dir_i, exist_ok=True) | ||
| env["PROMETHEUS_MULTIPROC_DIR"] = prom_dir_i | ||
| logger.info(f"Set PROMETHEUS_MULTIPROC_DIR for DP {i}: {prom_dir_i}") | ||
| setup_dp_prometheus_dir(i, env["PROMETHEUS_MULTIPROC_DIR"], env) |
There was a problem hiding this comment.
🔴 Bug 这里是在构造传给 subprocess.Popen 的子进程 env，但 setup_dp_prometheus_dir() 对 dp_id == 0 会把当前进程 base_dir 下已有 .db 文件移动到 dp0/。当前进程是 multi-api supervisor，不是 DP0 服务；这些文件可能来自父进程在导入 FastDeploy/初始化 metrics 时创建，移动后 DP0 子进程的 /metrics 会把 supervisor 的 .db 一起 collect，继续造成 DP0 指标污染。
建议修复方式：把“迁移已有 .db”限定在真正切换当前进程环境的场景，例如 env_dict is None 时才执行 DP0 move；或者给 helper 增加 move_existing 参数，并在 multi_api_server 调用时传 False。multi_api_server 只需要创建 base_dir/dp{i} 并写入子进程 env。
4 participants
Motivation
Fix metric interference when multiple data-parallel services run on one server by isolating Prometheus multiprocess files per DP rank.
Modifications
PROMETHEUS_MULTIPROC_DIRset during metrics initialization.setup_dp_prometheus_dir()to create per-DPdp{i}subdirectories and switch the target environment.LLMEngine/EngineServiceand whenmulti_api_serverstarts per-DP API server processes.Usage or Command
N/A
Accuracy Tests
N/A
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.